Statistical Guides for Literary Analysis: A Test

The existence of machine readable text makes possible the development of new techniques that assist the literary scholar in locating interesting passages of text. In this paper we explore in a preliminary manner the possibility of adapting techniques developed in the field of document retrieval to the full text context. As an alternative to the conventional Boolean logic based approaches, we consider techniques in which words are assigned weights automatically on the basis of how likely they are to distinguish useful passages from not useful passages, and in which passages containing the words with the largest weights are retrieved. In general, these techniques are iterative and take advantage of a user’s evaluation of initial retrievals. In the research reported in this paper we tested one stage of this process. Interesting passages (in our case, passages containing references to Charlemagne) were collected as a criterion corpus, and the weighting mechanism was used to assign weights to words on the basis of how well these words separated these passages from the rest of the text. The appropriateness of the weight assignment, based upon semantic considerations, is discussed, as well as new opportunities for research suggested by this study. key words: full text search, information retrieval, probabalistic retrieval, automatic information retrieval, Bayesian statistical methods: aplications to text search, weighted indexing, ARTFL, TLF, Charlemagne, Computer assisted literary research